Machine learning techniques that identify attribution of small signal stimulus in noisy response channels

ABSTRACT

An example apparatus includes a model generator to generate a learning model based on a correlation of stimulus data and response data, the learning model to predict user responses based on stimuli presented to the users in the channel or the sub-channel, the correlation indicative of stimuli contributing to user responses at a channel or a sub-channel level. The apparatus further includes an attribution engine to determine a media spend plan based on the learning model and a budget, the media spend plan including an allocation of the budget to stimuli corresponding to the channel or the sub-channel and a user interface to display the media spend plan to a user and update the media spend plan based on predictions of the learning model when the user adjusts the budget or allocations of the media spend plan in the user interface.

RELATED APPLICATIONS

This patent arises from a continuation of U.S. patent application Ser.No. 15/630,806, entitled “MACHINE LEARNING TECHNIQUES THAT IDENTIFYATTRIBUTION OF SMALL SIGNAL STIMULUS IN NOISY RESPONSE CHANNELS,” filedJun. 22, 2017, U.S. Pat. No. ______, which claims the benefit of U.S.Provisional Patent Application Ser. No. 62/356,092, entitled“IDENTIFYING SMALL SIGNAL MEDIA STIMULUS CORRELATIONS IN NOISY MEDIARESPONSE CHANNELS,” filed Jun. 29, 2016. U.S. patent application Ser.No. 15/630,806 and U.S. Provisional patent application Ser. No.12/430,978 are hereby incorporated by reference in their respectiveentireties.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF THE DISCLOSURE

The disclosure relates to the field of media spend optimization and moreparticularly to techniques for identifying small signal stimuluscorrelations in noisy response channels.

BACKGROUND

The technological capability availed by today's computing systems anddata communications systems (e.g., networks, the Internet) has greatlyimpacted the ability to communicate to a vast number of users.Technology permits implementing systems that communicate information,such as in the form of messages and notifications, to a vast numbers ofusers in order to elicit a desired response from the users. For example,a communications and management system may be built to intervene in thedaily routine of a diabetes patient in order to effectively manage thepatient's blood glucose levels. To this end, management systems may sendvarious types of notifications, at various times, to remind and inform auser so as to effectively manage the user for a desired outcome. Inaddition to transmitting notifications and messages to users, thesecommunication and management systems may also include a means to receiveuser responses in order to receive feedback as to whether the userperformed the desired task. For example, in a diabetes managementsystem, the system may record a response, using technology, thatindicates the user took their medication at a specific time of day. Theprocess of transmitting notifications and messages to users toeffectuate a user response, as well as receiving and recording thoseresponses constitute the basis for such a management system.

It is desirable to use such computing and networking capabilities togain a deeper understanding of the impact of event notifications onevents. For example, if a system propagates messages to a user to remindthe user to take their medication at specific times of the day, it isdesirable to understand what messages are effective at getting thepatient to properly take their medication. Since an event campaign maycomprise many different types of messages and notifications, it is notclear which messages or notifications were effective in getting thedesired response. As such, a system administrator may desire to knowwhat messages, sent when, and in what combinations, were most effectiveat eliciting the desired result. Given the capabilities of computing andnetworking, there are many mediums, or channels, that may be used todeliver messages to users. For example, messages may be delivered intoan environment, such as a house, using various modes of communications,such as Internet messages (e.g., display messages), email, text messageson mobile devices, notifications on television, telephone calls,announcement over audio/visual medium, etc. These systems may alsodeploy messages to users in order to elicit commercial activity from auser, such as conversions to purchase a product or service.

Various techniques for calculating the attribution of event campaignstimuli (e.g., event notifications) to responses have been considered.Such attribution calculations are largely enabled not only by thevoluminous online user activity data available (e.g., cookies, pixeltags, mobile tracking, etc.), but also by various offline data available(e.g., in-store purchase records, compliance records, etc.). However, in“noisy” response channels (e.g., organic search, surveys, compliancefeedback, etc.) that may include an aggregate response from multipleevent campaigns and event stimuli, legacy approaches have limitations.Such legacy approaches might provide attribution at a channel level(e.g., TV, radio, print, search, etc.), yet not provide attribution at amore granular sub-channel level. For example, a legacy approach mightattribute call center purchases to a TV infomercial providing thetoll-free number of the call center at the “TV” media channel level, yetnot identify more specific airings and related characteristics (e.g., TVstation, spot, campaign, creative, etc.) that contributed most to thepurchases. Techniques are needed to address the problem of determiningsub-channel stimulus attribution in aggregated response channels.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a partitioning for a system that implements techniquesfor identifying small signal media stimulus correlations in noisy mediaresponse channels, according to an embodiment.

FIG. 2A is a portfolio schematic showing multiple media stimuluschannels and response channels in environments for implementingtechniques for identifying small signal media stimulus correlations innoisy media response channels, according to some embodiments.

FIG. 2B is a correlation chart showing time-based and value-basedcorrelation techniques used identify small signal media stimuluscorrelations in noisy media response channels, according to someembodiments.

FIG. 3A presents a diagram illustrating stimulus signals and responsesignals for a TV media campaign.

FIG. 3B presents a schematic of a signal and data flow for implementingtechniques for identifying small signal media stimulus correlations innoisy media response channels

FIG. 4 depicts a subsystem used to identify small signal media stimuluscorrelations in noisy media response channels, according to someembodiments.

FIG. 5 is a flow chart showing steps used to identify small signal mediastimulus correlations in noisy media response channels, according tosome embodiments.

FIG. 6 depicts an environment in which embodiments of the presentdisclosure may operate.

FIG. 7 depicts an environment in which embodiments of the presentdisclosure may operate.

FIG. 8 is a portfolio schematic showing multiple media stimulus channelsand response channels in environments for implementing techniques foridentifying small signal media stimulus correlations in noisy mediaresponse channels, according to some embodiments.

FIG. 9 is a diagram depicting data structures used in systems foridentifying small signal media stimulus correlations in noisy mediaresponse channels, according to some embodiments.

FIG. 10 is a TV station attribution chart produced by systems thatidentify small signal media stimulus correlations in noisy mediaresponse channels, according to some embodiments.

FIG. 11A shows a user interface for establishing channel levelallocation of media spend in systems implementing techniques foridentifying small signal media stimulus correlations in noisy mediaresponse channels, according to some embodiments.

FIG. 11B shows a user interface for sub-channel level allocation ofmedia spend based on small signal media stimulus correlations in noisymedia response channels, according to some embodiments.

FIG. 12A and FIG. 12B depict block diagrams of computer systemcomponents suitable for implementing embodiments of the presentdisclosure.

DETAILED DESCRIPTION Overview

In one or more embodiments, the techniques described herein furtherdiscuss (6) calculating contribution values for the stimulus attributesequences based at least in part on the correlation coefficients, and(7) apportioning new stimulus based at least in part on the correlationcoefficients.

Definitions

Some of the terms used in this description are defined below for easyreference. The presented terms and their respective definitions are notrigidly restricted to these definitions—a term may be further defined bythe term's use within this disclosure.

-   -   The term “exemplary” is used herein to mean serving as an        example, instance, or illustration. Any aspect or design        described herein as “exemplary” is not necessarily to be        construed as preferred or advantageous over other aspects or        designs. Rather, use of the word exemplary is intended to        present concepts in a concrete fashion.    -   As used in this application and the appended claims, the term        “or” is intended to mean an inclusive “or” rather than an        exclusive “or”. That is, unless specified otherwise, or is clear        from the context, “X employs A or B” is intended to mean any of        the natural inclusive permutations. That is, if X employs A, X        employs B, or X employs both A and B, then “X employs A or B” is        satisfied under any of the foregoing instances.    -   The articles “a” and “an” as used in this application and the        appended claims should generally be construed to mean “one or        more” unless specified otherwise or is clear from the context to        be directed to a singular form.

Solutions Rooted in Technology

The appended figures corresponding discussion given herein providessufficient disclosure to make and use systems, methods, and computerprogram products that address the aforementioned issues with legacyapproaches. More specifically, the present disclosure provides adetailed description of techniques used in systems, methods, and incomputer program products for identifying small signal stimuluscorrelations in user response channels. Certain embodiments are directedto technological solutions for receiving electronically at a server datarecords corresponding to multiple user stimulus vectors and userresponse vectors, and generating correlation coefficients to model theinfluence of the user stimulus vectors on the user response vectors atsub-channel levels (e.g., campaign, station, spot, creative, etc.),which embodiments advance the relevant technical fields, as well asadvancing peripheral technical fields. The disclosed embodiments modifyand improve over legacy approaches. In particular, the herein-disclosedtechniques provide technical solutions that address the technicalproblems attendant to determining sub-channel user stimulus attributionin aggregated user response channels. Such technical solutions serve toreduce use of computer memory, reduce demand for computer processingpower, and reduce communication overhead needed. Specifically, theherein disclosed technical solutions may eliminate the computing andstorage resources required by legacy techniques in installing furtherdirect response measurement facilities associated with certain noisymedia response channels. Some embodiments disclosed herein usetechniques to improve the functioning of multiple systems within thedisclosed environments, and some embodiments advance peripheraltechnical fields as well. As one specific example, use of the disclosedtechniques and devices within the shown environments as depicted in thefigures provide advances in the technical field of high-performancecomputing as well as advances in various technical fields related todistributed storage.

Reference is now made in detail to certain embodiments. The disclosedembodiments are not intended to be limiting of the claims.

DESCRIPTIONS OF EXEMPLARY EMBODIMENTS

FIG. 1 depicts a partitioning 100 for a system that implementstechniques for identifying small signal media stimulus correlations innoisy user response channels. As an option, one or more instances ofpartitioning 100 or any aspect thereof may be implemented in the contextof the architecture and functionality of the embodiments describedherein. Also, the partitioning 100 or any aspect thereof may beimplemented in any desired environment.

As shown in FIG. 1, the stimulus data 172 is presented to the user group110 resulting in the response data 174. Generally, at least one responsemeasurement in the response data 174 is attempted for each stimulusassociated with the stimulus data 172. 2For example, an eventnotification in a user stimulus channel might be measured by a userresponse in the user response channel. As shown, a learning model 192may be formed using the stimulus data 172 and response data 174. Thelearning model 192 serves to predict a particular channel response froma particular channel stimulus. For example, if an event notificationfrom last Saturday and Sunday resulted in a user performing a specifictask, then the learning model 192 may predict that additional eventnotifications next Saturday and Sunday might result in the same userbehavior. One technique to train the learning model 192 uses a simulator193 and a model validator 194, as shown. The simulator 193 may providevarious subsets of the stimulus data 172 to the learning model 192 andthe responses predicted by the learning model 192 are compared by themodel validator 194 to the actual responses from the response data 174.The results of the comparison may be used to adjust the learning model192 such that a true attribution of response credit to a given stimulusand/or set of stimuli may be established. Such instances of attributiondata 176 may be stored in the measurement data store 126. Varioustechniques might be implemented in the learning model 192 to addressvarious attribution biases (e.g., channel saturation, etc.).

A simulated model 196 may further serve to determine a set of stimuliand associated set of predicted responses even when direct measurementsare not available. The simulated model 196 may be formed using anymachine learning techniques and/or the operations shown in FIG. 1.Specifically, the embodiment of FIG. 1 supports a technique wherevariations (e.g., mixes) of instances of the stimulus data 172 aredelivered by the simulator 193 to the learning model 192 to capturepredictions of the responses to a particular various eventnotifications. The full range of stimulus variations and associatedpredicted response variations form, in part, the simulated model 196.Such techniques may also serve to establish response and performancelimits. In some embodiments, a systems manager may use the simulator 193and the simulated model 196 to establish instances of eventapportionment data 178 (e.g., in the planning data store 127) used invarious types of campaigns.

As shown, the herein disclosed techniques solve such technical problemsby identifying small signal stimulus correlations in noisy responsechannels using, in part, a small signal correlation engine 182. Morespecifically, the small signal correlation engine 182 may receiveelectronic data records comprising the stimulus data 172 and electronicdata records comprising the response data 174 (e.g., noisy channelresponse data 175), and generate correlation coefficients each stimulussignal comprising the stimulus data 172. The correlation coefficientsmay indicate a correlation of the stimulus signals to any or all of theresponse signals comprising the response data 174. The correlationcoefficients effectively serve to “source separate” the multiplecontributors to a given response and/or set of responses. Thecorrelation coefficients and other associated model parameters may bedelivered to the learning model such that sub-channel attribution (e.g.,contribution values) of stimulus signals may be calculated. Suchtechniques shown in FIG. 1 and described herein may be applied tovarious channels where sub-channel (e.g., granular) stimulus attributionis required for accurate apportioning of event notification.

FIG. 2A is a portfolio schematic 2A00 showing multiple event stimuluschannels and response channels in environments for implementingtechniques for identifying small signal event stimulus correlations innoisy response channels. As an option, one or more instances ofportfolio schematic 2A00 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the portfolio schematic 2A00 or any aspectthereof may be implemented in any desired environment.

As shown, the portfolio schematic 2A00 includes three types ofcommunication mediums, namely communications medium₁ 207, communicationsmedium₂ 203, and communications medium₃ 206. Under each communicationsmedium are shown one or more channels for deliver of events.Communications medium₃ 207 comprises channels named CH1 208 and CH2 210.Communications medium₂ 203 comprises channels named CH3 212.Communications medium₃ 206 comprises distribution through CH4 226, CH5228, and CH6 230. More specifically, the portfolio includes events forcommunications medium₁ 207 during the evening 214, weekly series 216,and morning 218. The event portfolio also includes communicationsmedium₂ 203 notifications in the form of a sponsored public serviceannouncement 220, a sponsored event 222, and an opt-in event 224. Theevent portfolio also includes communications medium₃ 206 for eventsdelivered by mediums 227, a print media placement 229, and a publicbulletin board 231, as shown. The stimuli comprising such events may beviewed in more detail. For example, the events for communicationsmedium₁ 207 on station CH1 208 during the evening 214 may comprisecertain attributes characterized by a timestamp, a duration, aplacement, a creative, and/or other characteristics. In some cases, thesystems manager might want to know a particular airing that mostinfluenced a particular response.

As depicted in FIG. 2A, for each channel and sub-channel, there is oneor more stimuli (e.g., S1 , S2 , S3 , to SN) and a respective measuredresponse (e.g., R1 , R2 , R3 , to RN). As shown, there may be aone-to-one correspondence between a particular stimulus and itsresponse. For example, the communications medium₂ 207 evening event 214is depicted with stimulus S1 , and has an associated response R1 (e.g.,using a survey 232 metric). In some cases, certain sets of indirectresponse channels 240 may be associated with certain stimuli. Forexample, the response channels R9 , R10 , and R11 , comprising theindirect response channels 240, might be associated with the eventcampaign using the communications medium₁ 207. In this case, theresponse data from the indirect response channels 240 might be available(e.g., as response data 174) in aggregate, and/or include the influenceof instances of other stimuli 245 (e.g., not from communications medium₁207).

The event stimuli and user responses discussed herein are often formedas a sequential time-series (e.g., sequences) of data items (e.g.,attributes, values, measurements) representing various stimulusinstances (e.g., events) and various user response instances,respectively. An exemplary embodiment of data structures used for suchstimulus event sequences and user response measurement attributesequences is described in FIG. 9. For notational convenience, suchtemporal signals may be represented as stimulus vectors, such as thestimulus vector 202 ₁ (e.g., corresponding to instances in time ofstimulus S3 ), the stimulus vector 202 ₂ (e.g., corresponding toinstances in time of stimulus S7 ), and the response vector 204 (e.g.,corresponding to instances in time of response R3 ). As shown, astimulus or response vector may have many data items (e.g., stimulusvector 202 ₁ comprising multiple events, or could have as few as onedata item (e.g., stimulus vector 202 ₂ comprising a single eventcorresponding to event 227). In some cases, the time interval and/ortime stamp of certain stimuli and certain responses may be the same,such as shown in FIG. 2A (e.g., times T1, T2, T3, etc.). In other cases,the time interval and/or timestamp of certain stimuli and certainresponses may be different. Further, the time-series may be presented ina native time unit (e.g., weekly, daily) and may be apportioned over adifferent time unit. For example, stimulus S3 corresponds to a weeklyevent, even though the stimulus to be considered actually occurs daily(e.g., during the morning). The weekly stimulus spend may be apportionedto a daily stimulus occurrence. In some situations, the time unit in atime-series may be granular (e.g., minutes, seconds). When the intervalis short (e.g., seconds or minutes) for a given vector or collection ofvectors, a visualization plot of the vector or vectors over time mayappear as a continuous signal. Apportioning over time periods or timeunits may be performed using any known techniques. Vectors (e.g.,instances of stimulus vectors, instances of response vectors, etc.) maybe formed from any time-series in any time units and may be apportionedto another time-series using any other time units.

FIG. 2B is a correlation chart 2B00 showing time-based and value-basedcorrelation techniques used to identify small signal event stimuluscorrelations in noisy user response channels. As an option, one or moreinstances of correlation chart 2B00 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the correlation chart 2B00 or anyaspect thereof may be implemented in any desired environment.

A particular stimulus in a first event stimulus channel (e.g., S1 )might produce measured response results in a first user response channel(e.g., R1 ). Additionally, a stimulus in a first event stimulus channel(e.g., S1 ) might produce results, or lack of results, as given bymeasured results in a different response channel (e.g., R3 ). Suchcorrelation of results or lack of results may be automatically detected,and a scalar value representing the extent of correlation may bedetermined mathematically from any pair of vectors. In the discussionjust below, the correlation of a time-series response vector isconsidered with respect to a time-series stimulus vector. In some cases(e.g., noisy response channels with multiple stimulus channels), thecorrelation between stimulus and response may be one-to-many ormany-to-one. Correlations may be positive (e.g., the time-series datamoves in the same directions), or negative (e.g., the time-series datamoves in the opposite directions), or zero (e.g., no correlation). Thoseskilled in the art will recognize there are many known-in-the-arttechniques to correlate any pair of curves. For example, correlation maybe based on a distance function that determines a spatial distanceand/or a temporal distance between event attributes (e.g., data items)and response attributes (e.g., data items).

As shown, vector S1 is comprised of a series of changing values (e.g.,depicted by the regression-fitted function resulting in the curve 290).The response R1 is shown as curve 291. As may be appreciated, eventhough the curve 291 is not identical to the curve 290 (e.g., it hasundulations in the tail) the curve 291 is substantially value-correlatedto curve 290. Maximum value correlation 297 occurs when curve 291 istime-shifted by a Δt 292 amount of time relative to curve 290 (see theΔt 292 graduations on the time scale) and a time period of 2Δt isconsidered. For example, the Δt 292 might represent a delay in theresponse to a given stimulus. The amount of correlation (see discussioninfra) and amount of time shift may be automatically determined. Variousexamples of channel correlations are presented in Table 1 and in FIG.2B.

TABLE 1 Cross-correlation examples Stimulus Channel → Response channelDescription S1 → R2 (see curve 294) No correlation S1 → R3 (see curve295) Correlates if time shifted and attenuated S1 → R4 (see curve 296)Correlates if time shifted and amplified

In some cases, a correlation calculation may identify a negativecorrelation where an increase in a first stimulus channel causes adecrease in a second response channel. Further, in some cases, acorrelation calculation may identify an inverse correlation where alarge increase in a first channel causes a small increase in a secondchannel. In still further cases, there may be no observed correlation(e.g., see curve 294), or in some cases correlation is increased whenexogenous variables are considered (e.g., see curve R1 ^(E) 293).

In some cases a correlation calculation may hypothesize one or morecausation effects. And in some cases correlation conditions areconsidered when calculating correlation such that a priori knownconditions may be included (or excluded) from the correlationcalculations.

Also, as may be appreciated, there is no correlation to the showntime-series R2. The curve 295 is substantially value-correlated (e.g.,though scaled down) to curve 290, and is time-shifted by a second Δtamount of time relative to curve 290. The curve 296 is substantiallyvalue-correlated (e.g., though scaled up) to curve 290, and istime-shifted by a second Δt amount of time relative to curve 290.

In some cases, correlation parameters are provided to handle specificcorrelation cases. In one case, the correlation between two time-seriesmay be determined to a scalar value (e.g., Pearson's product-momentcoefficient) using Eq. 1.

$\begin{matrix}{r = \frac{{n{\sum{xy}}} - {( {\sum x} )( {\sum y} )}}{\sqrt{{n( {\sum x^{2}} )} - ( {\sum x} )^{2}}\sqrt{{n( {\sum y^{2}} )} - ( {\sum y} )^{2}}}} & (1)\end{matrix}$

where:

x represents components of a first time-series,

y represents components of a second time-series, and

n is the number of {x, y} pairs.

Other correlation techniques are possible, and a systems manager mightprovide an indication and parameters associated with such alternativecorrelations. For example, parameters known as “AR”, “MA”, and “BW” areused in an autoregressive integrated moving average model (ARIMA). Otherparameters such as “FF” to characterize a forgetting factor, and “L” tocharacterize a length duration of the response variables may be includedin correlation calculations.

In some cases, while modeling a time-series, not all the scalar valuesin the time-series are weighted equally. For example, more recenttime-series data values found in the historical data may be given ahigher weight as compared to older ones. Various shapes of weights tooverlay a time-series are possible, and one exemplary shape is the shapeof an exponentially decaying model.

Such correlation techniques may be used by the herein disclosedtechniques for identifying small signal event stimulus correlations innoisy response channels. An illustration of such a noisy responsechannel is shown in FIG. 3A.

FIG. 3A presents a diagram 3A00 illustrating stimulus signals andresponse signals for a TV media campaign. As an option, one or moreinstances of diagram 3A00 or any aspect thereof may be implemented inthe context of the architecture and functionality of the embodimentsdescribed herein. Also, the diagram 3A00 or any aspect thereof may beimplemented in any desired environment.

The diagram 3A00 illustrates a time-series visualization of the variousstimulus signals (e.g., vectors) and response signals (e.g., vectors)associated with a campaign. Specifically, FIG. 3A depicts an eventscampaign having multiple event stimulus signals and a noisy responsechannels that may use the herein disclosed techniques to identifycorrelations between such small signal stimulus and the noisy responsechannels. More specifically, in reference to the portfolio schematic2A00 in FIG. 2A, the stimulus signals for the shown campaign comprisestimulus vectors associated with the evening event 214, the weeklyseries 216 spend, and the morning event 218. The diagram 3A00 furthershows stimulus vectors associated with an infomercial 302 and a healthnetwork 304. As earlier mentioned, the stimulus vectors each represent asequential time-series of data items corresponding to various stimulusinstances. For example, the pulses illustrated in FIG. 3A may representcertain times and durations of airings in the respective media spendsub-channels.

Also represented in diagram 3A00 are signal waveforms associated withthe other stimuli 245 from FIG. 2A, and certain instances of responsevectors 312 measured and collected for the campaign. The signalwaveforms associated with the other stimuli 245 may represent stimulusvectors not included in the subject campaign, yet influence the responsevectors 312. Further, the signal waveforms associated with the responsevectors 312 may represent response vectors from certain responsechannels identified as being associated with the subject campaign. Forexample, the response vectors 312 might comprise responses from directresponse channels (e.g., R1, R2 and R3, etc.) and/or from indirectresponse channels, such as the indirect response channels 240 (e.g., R9,R10 and R11, etc.).

As shown, the response vectors 312, when aggregated, may comprise anoisy signal, presenting a challenge to identifying respectivecorrelations between each sub-channel stimuli and the response data. Theexample in diagram 3A00 is merely one example. Some system campaigns mayhave a higher number of stimulus vectors and response vectors thanshown. In such cases, the technological solution provided by the hereindisclosed techniques may be implemented. Such an implementation isdescribed in more detail in FIG. 3B.

FIG. 3B presents a schematic 3B00 of a signal and data flow forimplementing techniques for identifying small signal event stimuluscorrelations in noisy user response channels. As an option, one or moreinstances of schematic 3B00 or any aspect thereof may be implemented inthe context of the architecture and functionality of the embodimentsdescribed herein. Also, the schematic 3B00 or any aspect thereof may beimplemented in any desired environment.

Shown in schematic 3B00 are techniques for determining sub-channel eventstimulus attribution in aggregated and/or noisy user response channels(e.g., the response vectors 312). Specifically, FIG. 3B describesfurther data flow and operations details pertaining to the small signalcorrelation engine 182 disclosed herein. Specifically, the small signalcorrelation engine 182 is shown to receive electronic data recordscomprising stimulus data 172 and response data 174. In exemplary casesdescribed herein, the stimulus data may be associated with multiplestimulus channels (e.g., S1 to SN) producing a plurality of stimulusvectors 321. More specifically, the stimulus vectors 321 may comprisestimulus touchpoint attribute sequences describing the touchpointsevents experienced by one or more audiences. Further, the response data174 may be associated with multiple measured response channels (e.g., R1to RN) producing a plurality of response vectors that may be combined inan aggregate response vector 322. More specifically, the aggregateresponse vector 322 may comprise response measurement attributesequences capturing the user interactions (e.g., specific task(s)performed by certain audience members). Given such instances of thestimulus vectors 321 and the aggregate response vector 322, the smallsignal correlation engine 182 may be used to identify correlations ofsmall signal event stimulus (e.g., stimulus vectors 321) to theresponses in noisy user response channels (e.g., comprising theaggregate response vector 322).

In some embodiments, the small signal correlation engine 182 selects aportion of the aggregate response vector 322 using a filter 346. Forexample, a certain portion of responses might be selected for processing(e.g., correlation) based in part on the response geography (e.g.,geographical origin of the responses), the IP address of the respondent,the response time, the response channel, and/or other filtercharacteristics. The transfer function h(n) of the filter 346 may alsobe a vector that will serve to include a first portion of the responsevectors comprising the aggregate response vector 322 in a filteredaggregate response vector 324, and exclude a second portion of theresponse vectors comprising the aggregate response vector 322 from thefiltered aggregate response vector 324. The purpose of such filteringmay vary. For example, the filter 346 may be used to partition thecorrelation operation to balance the use of computing resources. Asanother example, the filter 346 may be used to reduce the time to reacha correlation solution for a given instance of the filtered aggregateresponse vector 324 and a given set of stimulus vectors 321.

A correlator 348 receives the stimulus vectors 321 and the filteredaggregate response vector 324 and calculates a corresponding set ofstimulus vector coefficients k, 326 (e.g., represented by the magnitudeof the bar chart shown in FIG. 3B). In one or more embodiments, thestimulus vector coefficients k_(i) 326 may be calculated by loopingthrough each stimulus vector comprising the stimulus vectors 321 andcorrelating the selected stimulus vector to the filtered aggregateresponse vector 324 (see operation 350). The relationship between thestimulus vectors 321, the stimulus vector coefficients k_(i) 326, andthe filtered aggregate response vector 324 may be described in acorrelation model 330. In some embodiments, the stimulus vectorcoefficients k_(i) 326 and a certain set of model input parameters 328may be delivered to the learning model 192. For example, the stimulusvector coefficients k_(i) 326 may be used to calculate sub-channelattribution (e.g., contribution values) by iterating over each of thestimulus vectors 321. In some cases, correlations and attributions forcertain events comprising each of the stimulus vectors 321 may also becalculated.

FIG. 4 depicts a subsystem 400 used to identify small signal eventstimulus correlations in noisy user response channels. As an option, oneor more instances of subsystem 400 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the subsystem 400 or any aspectthereof may be implemented in any desired environment.

As shown, events may be generated by the system administrators 142 as aset of stimulus data 172 to the audience 110 to produce a set ofresponse data 174 captured by the data management server 138. Themeasurement server 132 may receive electronic data records associatedwith the stimulus data 172 and response data 174 (see operation 402).The stimulus data 172 and response data 174 may be stored in one or morestorage devices 120 (e.g., stimulus data store 124, response data store125, etc.). The stimulus data 172 and response data 174 may further beused by the small signal correlation engine 182 to identify small signalevent stimulus correlations in noisy user response channels.Specifically, the small signal correlation engine 182 may filter theresponse data 174 (see operation 404) and generate stimulus vectorcorrelation coefficients (see operation 406). The stimulus vectorcorrelation coefficients and other model parameters may be used by amodel generator 410 to generate a learning model (see operation 412) andgenerate a simulated model (see operation 414).

As shown, the apportionment server 134 may receive the model parametersfrom the model generator 410 in the measurement server 132 (seeoperation 422) and enable an attribution engine 430 to calculatestimulus contribution values (see operation 432). For example, in someembodiments, the attribution engine 430 uses the stimulus vectorcorrelation coefficients, generated by the small signal correlationengine 182, to calculate the granular sub-channel stimulus contributionvalues. The attribution engine 430 may deliver performance reports (seeoperation 434) associated with the stimulus data 172, response data 174,and/or various marketing campaigns and/or other data. Further, theattribution engine 430 may enable (e.g., for system managers) mediaspend planning and/or optimization based in part on the data and/oroperations availed by the subsystem 400.

The subsystem 400 presents merely one partitioning. The specific exampleshown where the measurement server 132 comprises the small signalcorrelation engine 182 and the model generator 410, and where theapportionment server 134 comprises the attribution engine 430 is purelyexemplary, and other partitioning is reasonable, and the partitioningmay be defined in part by the volume of empirical data. In some cases, adatabase engine may serve to perform calculations (e.g., within, or inconjunction with a database engine query). A technique for identifyingsmall signal event stimulus correlations in noisy response channelsimplemented in such systems, subsystems, and partitionings is shown inFIG. 5.

FIG. 5 is a flow chart 500 showing steps used to identify small signalstimulus correlations in noisy user response channels. As an option, oneor more instances of flow chart 500 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the flow chart 500 or any aspectthereof may be implemented in any desired environment.

The flow chart 500 presents one embodiment of certain steps foridentifying small signal event stimulus correlations in noisy userresponse channels. In one or more embodiments the steps and underlyingoperations shown in the flow chart 500 may be executed by themeasurement server 132 and apportionment server 134 disclosed herein. Asshown, the flow chart 500 commences with receiving stimulus data andresponse data (see step 502). For example, the stimulus data maycomprise a plurality of stimulus vectors. Also, the response data maycomprise a plurality of response measurement vectors, a portion of whichmay be aggregated. The response data may be filtered (see step 504). Forexample, a certain portion of the response vectors comprising theresponse data may be included in the filtered response data based inpart on geographical origin of the responses, the IP address of therespondent, the time of the response, the response channel, and/or otherfilter characteristics. A set of stimulus vector correlationcoefficients for each stimulus vector comprising the received stimulusdata may be calculated (see step 506). In one or more embodiments, step506 may be implemented with a FOR loop comprising: (1) select a stimulusvector from the plurality of stimulus vectors in the stimulus data, (2)correlate the selected stimulus vector to the filtered response data,and (3) exit the FOR loop when all stimulus vectors have beencorrelated.

The flow may continue with generating a learning model (see step 512)and generating a simulated model (see step 514) as described herein. Thecorrelation coefficients and model parameters may be collected (see step522) and used to calculate stimulus contribution values (see step 532).Further, performance reports associated with the stimulus data, responsedata, stimulus correlations, and/or various campaigns and/or other datamay be delivered (see step 534). Also, media spend planning and/oroptimization (e.g., for a systems manager) may be enabled (see step536).

Internet of Things System Embodiments

FIG. 6 depicts an environment 600 in which embodiments of the presentdisclosure can operate. As an option, one or more instances ofenvironment 600 or any aspect thereof may be implemented in the contextof the architecture and functionality of the embodiments describedherein. Also, the environment 600 or any aspect thereof may beimplemented in any desired environment.

The present invention has application for systems that utilize theInternet of Things (IOT). For these embodiments, systems communicate toenvironments, such as a home environment, to employ event campaigns thatuse stimuli to effectuate desired user responses. Specifically, devicesmay be placed in the home to both communicate event messages ornotifications as well as receive responses, either responses gathered bysensing users or by direct input to electronic devices by the users.Embodiments for implementing the present invention in such anenvironment are shown in FIG. 6.

The shown environment 600 depicts a set of users (e.g., user 605 ₁, user605 ₂, user 605 ₃, user 605 ₄, user 605 ₅, to user 605 _(N)) comprisingan audience 610 that might be targeted by one or more event sponsors 642in various event campaigns. The users may view a plurality of eventnotifications 653 on a reception device 609 (e.g., desktop PC, laptopPC, mobile device, wearable, television, radio, etc.). The eventnotifications 653 can be provided by the event sponsors 642 through anyof a plurality of channels 746 in the wired environment (e.g., desktopPC, laptop PC, mobile device, wearable, television, radio, print, etc.).Stimuli from the channels 646 comprise instances of touchpoints 600experienced by the users. As an example, a TV spot may be viewed on acertain TV station (e.g., touchpoint T1), and/or a print message (e.g.,touchpoint T2) in a magazine. Further, the stimuli channels 746 mightpresent to the users a banner ad on a mobile browser (e.g., touchpointT3), a sponsored website (e.g., touchpoint T4), and/or an eventnotification in an email message (e.g., touchpoint T5). The touchpoints600 can be described by various touchpoint attributes, such as data,time, campaign, event, geography, demographics, impressions, cost,and/or other attributes.

According to one implementation, an IOT analytics platform 630 canreceive instances of stimulus data 672 (e.g., stimulus touchpointattributes, etc.) and instances of response data 674 (e.g., responsemeasurement attributes, etc.) via network 612, describing, in part, themeasured responses of the users to the delivered stimulus (e.g.,touchpoints 660). The measure responses are derived from certain userinteractions as sensed in the home (e.g., detector 604, sensor/infraredsensor 606, or monitoring device 611) or transmitted by the user (e.g.,mobile device 611, etc.) performed by certain users and can be describedby various response attributes, such as data, time, response channel,event, geography, revenue, lifetime value, and other attributes. Athird-party data provider 648 can further provide data (e.g., userbehaviors, user demographics, cross-device mapping, etc.) to the IOTanalytics platform 630. The collected data and any associated generateddata can be stored in one or more storage devices 620 (e.g., stimulusdata store 624, response data store 625, measurement data store 626,planning data store 627, audience data store 628, etc.), which are madeaccessible by a database engine 636 (e.g., query engine, resultprocessing engine, etc.) to a measurement server 632 and anapportionment server 634. Operations performed by the measurement server632 and the apportionment server 634 can vary widely by embodiment. Asan example, the measurement server 632 can be used to analyze certaindata records stored in the stimulus data store 624 and response datastore 625 to determine various performance metrics associated with anevent campaign, storing such performance metrics and related data inmeasurement data store 626. Further, for example, the apportionmentserver 634 may be used to generate event campaign plans and associatedevent spend apportionment, storing such information in the planning datastore 627.

Marketing Attribution Embodiments

FIG. 7 depicts an environment 700 in which embodiments of the presentdisclosure can operate. As an option, one or more instances ofenvironment 700 or any aspect thereof may be implemented in the contextof the architecture and functionality of the embodiments describedherein. Also, the environment 700 or any aspect thereof may beimplemented in any desired environment.

The present invention has application for marketing attribution systems.For these embodiments, marketing analytics platforms communicate toconsumers to employ marketing campaigns that use stimuli (e.g.,advertisements) to effectuate desired user responses, such asconversions. Embodiments for implementing the present invention in suchan environment are shown in FIG. 7.

The shown environment 700 depicts a set of users (e.g., user 705 ₁, user705 ₂, user 705 ₃, user 705 ₄, user 705 ₅, to user 705 _(N)) comprisingan audience 710 that might be targeted by one or more advertisers 742 invarious marketing campaigns. The users can view a plurality of adcontent 723 on a computing device (e.g., desktop PC 704, laptop PC 706,mobile device 708, wearable 709, etc.). The plurality of ad content 723can be provided by the advertisers 742 through any of a plurality ofonline channels 746 (e.g., online display, search, mobile ads, etc.)and/or a plurality of offline channels 744 (e.g., TV, radio, print,etc.). Stimuli from the online channels 746 and offline channels 744comprise instances of touchpoints 760 experienced by the users. As anexample, a TV spot 751 viewed on a certain TV station (e.g., touchpointT1), and/or a print ad 752 (e.g., touchpoint T2) in a magazine might bedelivered through the offline channels 744. Further, the online channels746 might present to the users a banner ad 753 on a mobile browser(e.g., touchpoint T3), a product website 754 (e.g., touchpoint T4),and/or a digital coupon 755 in an email message (e.g., touchpoint T5).The touchpoints 760 can be described by various touchpoint attributes,such as data, time, campaign, product, geography, demographics,impressions, cost, and/or other attributes.

According to one implementation, a marketing analytics platform 730 canreceive instances of stimulus data 772 (e.g., stimulus touchpointattributes, etc.) and instances of response data 774 (e.g., responsemeasurement attributes, etc.) via network 712, describing, in part, themeasured responses of the users to the delivered stimulus (e.g.,touchpoints 760). The measure responses are derived from certain userinteractions (e.g., call to call center, click on banner ad, purchaseonline, etc.) performed by certain users and can be described by variousresponse attributes, such as data, time, response channel, productpurchased, geography, revenue, lifetime value, and other attributes. Athird-party data provider 748 can further provide data (e.g., userbehaviors, user demographics, cross-device device mapping, etc.) to themarketing analytics platform 730. The collected data and any associatedgenerated data can be stored in one or more storage devices 720 (e.g.,stimulus data store 724, response data store 725, measurement data store726, planning data store 727, audience data store 728, etc.), which aremade accessible by a database engine 736 (e.g., query engine, resultprocessing engine, etc.) to a measurement server 732 and anapportionment server 734. Operations performed by the measurement server732 and the apportionment server 734 can vary widely by embodiment. Asan example, the measurement server 732 can be used to analyze certaindata records stored in the stimulus data store 724 and response datastore 725 to determine various performance metrics associated with amarketing campaign, storing such performance metrics and related data inmeasurement data store 726. Further, for example, the apportionmentserver 734 can be used to generate marketing campaign plans andassociated marketing spend apportionment, storing such information inthe planning data store 727.

Various techniques for calculating such performance metrics and spendapportioning have been considered. However, in “noisy” media responsechannels that might include an aggregate response from multiplemarketing campaigns and media stimuli, legacy approaches havelimitations. For example, user 705 ₁ in environment 700 might experienceall of the touchpoints 760 and then transition from a first engagementstate (e.g., no product interest) to a second engagement state (e.g.,conversion, such as a product purchase). Legacy approaches might be ableto measure the influence (e.g., contribution value) of certain stimulion the conversion of user 705 ₁ at a channel level (e.g., TV, radio,print, search, etc.), yet not discern the attribution at a more granularsub-channel level. For example, a legacy approach might not identify theTV spot 751 as the leading contributor to the conversion of user 705 ₁.The herein disclosed systems and techniques address such limitationsassociated with legacy approaches. One embodiment of such systems wasdescribed above in conjunction with FIG. 1.

FIG. 8 is a portfolio schematic 2A00 showing multiple media stimuluschannels and response channels in environments for implementingtechniques for identifying small signal media stimulus correlations innoisy media response channels. As an option, one or more instances ofportfolio schematic 800 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the portfolio schematic 800 or any aspectthereof may be implemented in any desired environment.

As shown, the portfolio schematic 800 includes three types of mediachannels, namely TV 807, radio 803, and print media 206. Under eachmedia channel are shown one or more sub-channel spends. TV 807 spendscomprise stations named ST1 808 and ST2 810. Radio 803 spends comprise astation named KVIQ 812. Print media 806 spends comprise distributionthrough mail 826, magazine 828, and printed coupon 830. Morespecifically, the media portfolio includes spends for TV 807 during theevening news 814, weekly series 816, and morning show 818. The mediaportfolio also includes radio 803 spends in the form of a sponsoredpublic service announcement 820, a sponsored shock jock spot 822, and acontest 824. The media portfolio also includes print media 806 spendsfor a direct mailer 827, a coupon placement 829, and an in-store coupon831, as shown. The stimuli comprising such spends can be viewed in moredetail. For example, the spends for TV 807 on station ST1 808 during theevening news 814 can comprise certain airings characterized by atimestamp, a duration, a placement (e.g., spot, show playing, etc.), acreative, and/or other characteristics. In some cases, the marketingmanager might want to know a particular airing that most influenced aparticular response.

As depicted in FIG. 8, for each media channel and sub-channel, there isone or more stimuli (e.g., S1 , S2 , S3 , to SN) and a respectivemeasured response (e.g., R1 , R2 , R3 , to RN). As shown, there can be aone-to-one correspondence between a particular stimulus and itsresponse. For example, the TV 807 evening news 814 spot is depicted withstimulus S1, and has an associated response R1 (e.g., using a Nielsenshare 832 metric). Also, response R1 is measured by calls to a callcenter 834, response R4 is measured by a number of calls 836, andresponse RN is measured by a number of in-store purchases 838. In somecases, certain sets of indirect response channels 840 can be associatedwith certain stimuli. For example, the response channels R9 , R10 , andR11 (e.g., paid search 842, website 843, organic search 844,respectively) comprising the indirect response channels 840 might beassociated with the campaign and spend using the TV 807 channel. In thiscase, the response data from the indirect response channels 840 might beavailable (e.g., as response data 174) in aggregate, and/or include theinfluence of instances of other stimuli 845 (e.g., not from TV 807).

The stimuli and responses discussed herein are often formed as asequential time-series (e.g., sequences) of data items (e.g.,attributes, values, measurements) representing various stimulusinstances (e.g., touchpoints) and various response instances,respectively. An exemplary embodiment of data structures used for suchstimulus touchpoint attribute sequences and response measurementattribute sequences is described in FIG. 9. For notational convenience,such temporal signals can be represented as stimulus vectors, such asthe stimulus vector 802 ₁ (e.g., corresponding to instances in time ofstimulus S3 ), the stimulus vector 802 ₂ (e.g., corresponding toinstances in time of stimulus S7 ), and the response vector 804 (e.g.,corresponding to instances in time of response R3 ). As shown, astimulus or response vector can have many data items (e.g., stimulusvector 802 ₁ comprising multiple airings on the evening news 814), orcould have as few as one data item (e.g., stimulus vector 802 ₂comprising a single event corresponding to the direct mailer 827). Insome cases, the time interval and/or time stamp of certain stimuli andcertain responses can be the same, such as shown in FIG. 8 (e.g., timesT1, T2, T3, etc.). In other cases, the time interval and/or timestamp ofcertain stimuli and certain responses can be different. Further, thetime-series can be presented in a native time unit (e.g., weekly, daily)and can be apportioned over a different time unit. For example, stimulusS3 corresponds to a weekly spend for the “Morning Show”, even though thestimulus to be considered actually occurs daily (e.g., during the“Morning Show”). The weekly stimulus spend can be apportioned to a dailystimulus occurrence. In some situations, the time unit in a time-seriescan be granular (e.g., minutes, seconds). When the interval is short(e.g., seconds or minutes) for a given vector or collection of vectors,a visualization plot of the vector or vectors over time can appear as acontinuous signal. Apportioning over time periods or time units can beperformed using any known techniques. Vectors (e.g., instances ofstimulus vectors, instances of response vectors, etc.) can be formedfrom any time-series in any time units and can be apportioned to anothertime-series using any other time units.

FIG. 9 is a diagram 600 depicting data structures used in systems foridentifying small signal media stimulus correlations in noisy mediaresponse channels. As an option, one or more instances of diagram 900 orany aspect thereof may be implemented in the context of the architectureand functionality of the embodiments described herein. Also, the diagram900 or any aspect thereof may be implemented in any desired environment.

As discussed herein, certain stimulus data (e.g., stimulus data 172) andcertain response data (e.g., response data 174) may be used in thedisclosed techniques and systems for identifying small signal mediastimulus correlations in noisy media response channels. The diagram 900depicts exemplary examples of structures for organizing, storing,accessing, analyzing, transmitting, or otherwise using such stimulusdata and response data. Specifically shown are a stimulus data structure910 and a response data structure 920. The stimulus data structure 910is represented in diagram 900 as a table with rows corresponding to astimulus vector and/or a data item (e.g., touchpoint, TV spot airingevent) included in a stimulus vector, and columns corresponding toattributes associated with the vectors and/or events. Specifically, therows in stimulus data structure 910 may comprise certain instances ofstimulus touchpoint attribute sequences 912 (e.g., comprising one ormore stimulus vectors). For example, the first row shown in the stimulusdata structure 910 corresponds to a 30-second “Creative 1” (e.g.,commercial for “Product 1”) for “Campaign 1” placed on “ABC” on“09/28/2013”. The timestamp for the spot is described as “at least up tothe minute” of “11:56:02” since an exact air time might not beguaranteed. The stimulus data structure 910 further captures a geography(e.g., Nielsen designated market area or DMA), a demography (e.g.,“Females 18-55”) and various impression metrics for the vector dataitem. For example, the impression data in the foregoing example indicatethat 1,234,567 impressions (e.g., individual and/or audience views) wereserved and the associated cost was $10,000. Other stimulus vectorsand/or data items are shown and/or are possible in the stimulus datastructure 910.

The response data structure 920 is represented in diagram 900 as a tablewith rows corresponding to a response vector and/or a data item (e.g.,paid search response) included in a response vector, and columnscorresponding to attributes associated with the vectors and/or events.Specifically, the rows in response data structure 920 may comprisecertain instances of response measurement attribute sequences 922 (e.g.,comprising one or more response vectors). For example, the second rowshown in the response data structure 920 corresponds to a “Paid Search”response using “Keyword 1” that resulted in a purchase of “Product 2” on“09/28/2013”. The timestamp for the response is described as “at leastup to the minute” of “11:57:30” since an exact search time might not berecorded. The response data structure 920 further captures a geography(e.g., Nielsen DMA) and various response metrics for the vector dataitem. For example, the response data in the foregoing example indicatethat $35.60 in revenue was generated from the response, and a lifetimevalue (LTV) of $1,256 was attributed to the respondent. Other keyperformance indicators may be included in the response data structure920, such as site (e.g., URL) visits, content “likes”, text messages,and other indicators. Other response vectors and/or data items are shownand/or are possible in the response data structure 920.

The herein disclosed techniques may be applied to stimulus data andresponse data organized in the stimulus data structure 910 and theresponse data structure 920, respectively, to identify small signalmedia stimulus correlations in noisy media response channels. Further,the small signal media stimulus correlations and other informationderived from the data structures in diagram 900 may be used to calculatethe influence (e.g., contribution value) a given stimulus vector has ona given response (e.g., conversion). One embodiment of such stimulusattribution results is shown in FIG. 10.

FIG. 10 is a TV station attribution chart 1000 produced by systems thatidentify small signal media stimulus correlations in noisy mediaresponse channels. As an option, one or more instances of TV stationattribution chart 1000 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the TV station attribution chart 1000 or anyaspect thereof may be implemented in any desired environment.

The TV station attribution chart 1000 is merely one example ofattribution calculations enabled, in part, by the herein disclosedtechniques for identifying small signal media stimulus correlations innoisy media response channels. Specifically, the TV station attributionchart 1000 illustrates the contribution values 1004 corresponding to oneor more stimulus vectors on each of the TV stations 1002. Suchsub-channel attribution in noisy response channels, such as thoseassociated with TV marketing, is enabled, in part, by the hereindisclosed techniques. For example, the contribution values 1004 mightcorrespond to the magnitude of the correlation coefficient generated bythe herein disclosed techniques for each of the TV stations 1002, suchcorrelation coefficients measuring the influence a certain spot orsequence of spots airing on a given station had on the purchase of aproduct.

As shown in the example of FIG. 10, the contribution values range from 0to nearly 16, and might represent an attribution ratio measurement. Forexample, one such ratio measurement compares the baseline conversionrate in a given segment to the conversion rate attributed to a givenstimulus. Other attribution and related metrics are possible. Forexample, the stimulus media spend in the stimulus data structure 910,and/or the response revenue and/or LTV included in the response datastructure 920, may be used to calculate other performance metrics, suchas cost per acquisition (CPA), and other performance metrics. Suchmetrics may be used (e.g., by a marketing manager) to plan and/oroptimize media campaigns and media spend allocations. User interfacesfor performing such tasks are discussed in FIG. 11A and FIG. 11B.

FIG. 11A shows a user interface 1100 for establishing channel levelallocation of media spend in systems implementing techniques foridentifying small signal media stimulus correlations in noisy mediaresponse channels. As an option, one or more instances of user interface1100 or any aspect thereof may be implemented in the context of thearchitecture and functionality of the embodiments described herein.Also, the user interface 1100 or any aspect thereof may be implementedin any desired environment.

The user interface 1100 may be invoked to show default conditions and/orto show the apportionments across the channels of a given marketingcampaign and/or media portfolio that corresponds to stimulus attributioncalculations enabled, in part, by the herein disclosed techniques. Byinteracting with the user interface 1100 a marketing manager (or anyuser) may reapportion spending over the channels in the media portfolioand view the results (e.g., using simulation, etc.) of suchreapportioning in real time. Specifically, and as shown, a marketingmanager may enter a budget amount using a budget field 1102 ₁ or byusing a budget allocation slider 1104. A default value for a budget maybe determined via a calculation that chooses a mid-point between auser-defined minimum budget and user-define maximum budget. Suchuser-defined budget points may be defined in a different interface view(e.g., the “Configuration Set Properties” tab). In some cases, thebudget might be “unknown”. In such a case, a default budget isdetermined. One approach to determining a default budget (e.g., aminimum budget) is to iterate to sum all of the minimum spend values asgiven through the entire portfolio, and use that value.

Responsive to a change in the budget amount, the system displays anallocation. The allocation may be displayed as a percent of the budget(e.g., using sliders or other display components to show an arrangementof channel allocation indications 1110 ₁), or the allocation may bedisplayed in the units of the budget (e.g., in dollars, as shown incolumn 1112). When allocations are established, the marketing managermay invoke activities that emulate and/or perform a simulation byclicking a simulate button 1114 ₁. Such activities serve to determine orpredict the effect that the selected allocations might have on theresponse of the media portfolio. In some cases, a user might want toreturn the selected channel allocations to the default and/orrecommended apportionment. In such a case, the user may interact with adisplay component to reset allocation (e.g., using reset allocationsbutton 1116 ₁). Any of a variety of known-in-the art techniques may beused to prevent unwanted overwriting of user values may be employedduring user interaction.

Sample Use Model

A use case proceeds a follows:

-   -   A user clicks on the reset allocations button 1116 ₁ to reset        the displayed values to depict values corresponding to stimulus        attribution calculations enabled, in part, by the herein        disclosed techniques.    -   A user clicks on the simulate button 1114 ₁. That action        initiates activities to determine response metric numbers for        the combination of the total budget and channel allocations as        specified by the user. If the total of all channel allocations        do not sum up to 100%, the user is prompted to further        reapportion spending percentages to reach 100%. The channel        allocations may be reapportioned to total 100% using any known        technique.    -   A user interacts with one or more channel allocation indications        1110 ₁ (e.g., using the per-channel allocation sliders). The        shown interface supports channel allocation indications to be        set to a value between the minimum budget and the maximum budget        as defined in the configuration set.

As shown in the channel allocation indications 1110 ₁ of the userinterface 1100, media spend allocations may be adjusted at the channellevel (e.g., TV, Print, Direct Mail, etc.). In some cases, the marketingmanager desires to allocate spending at a more granular level. Forexample, for the marketing manager might want to further optimize the19.8% of spend in the TV channel to specific stations and/or specificspots (e.g., airings, etc.). The herein disclosed techniques enableattributions at a sub-channel level by identifying small signal mediastimulus correlations in noisy media response channels (e.g., TV). Suchcorrelations and attributions may then be used to enable media spendapportioning as shown in FIG. 11B.

FIG. 11B shows a user interface 1101 for sub-channel level allocation ofmedia spend based on small signal media stimulus correlations in noisymedia response channels. As an option, one or more instances of userinterface 1101 or any aspect thereof may be implemented in the contextof the architecture and functionality of the embodiments describedherein. Also, the user interface 1101 or any aspect thereof may beimplemented in any desired environment.

The user interface 1100 shown in FIG. 11B comprises interface componentsdescribed in FIG. 11A. Specifically, instances of a budget field 1102 ₂,channel allocation indicators 1110 ₂, a reset allocations button 1116 ₂,and a simulate button 1114 ₂ are shown. As presented by the channelallocation indications 1110 ₂ and specified in a “% Spend” column 1113,certain allocations have been selected at the channel level (e.g.,TV=22.0%, Direct Mail=0.2%, Display Ads=4.0%, Paid Keyword Search=13.2%,and Other=18%). As described herein, the marketing manager may furtheruse the user interface 1101 to select certain instances of sub-channelallocation 1120. Specifically, the herein disclosed techniques mayenable stimulus-response correlation and attribution at a sub-channellevel (e.g., stimulus vector level, stimulus vector event level,stimulus vector data item level, etc.) stimulus such that media spendallocation scenarios may be simulated and planned. For example, asshown, the marketing manager may allocate the TV media spend to specificstations (e.g., CH 5499, CH HD 3999, CH HD 3400, CH 5500) based in parton the attribution resulting from correlations of the small signal mediastimulus associated with such stations to the response data receivedfrom noisy media response channels.

Additional System Architecture Examples

FIG. 12A depicts a diagrammatic representation of a machine in theexemplary form of a computer system 10A00 within which a set ofinstructions, for causing the machine to perform any one of themethodologies discussed above, may be executed. In alternativeembodiments, the machine may comprise a network router, a networkswitch, a network bridge, Personal Digital Assistant (PDA), a cellulartelephone, a web appliance or any machine capable of executing asequence of instructions that specify actions to be taken by thatmachine.

The computer system 10A00 includes one or more processors (e.g.,processor 1002 ₁, processor 1002 ₂, etc.), a main memory comprising oneor more main memory segments (e.g., main memory segment 1004 ₁, mainmemory segment 1004 ₂, etc.), one or more static memories (e.g., staticmemory 1006 ₁, static memory 1006 ₂, etc.), which communicate with eachother via a bus 1008. The computer system 10A00 may further include oneor more video display units (e.g., display unit 1010 ₁, display unit1010 ₂, etc.), such as an LED display, or a liquid crystal display(LCD), or a cathode ray tube (CRT). The computer system 10A00 may alsoinclude one or more input devices (e.g., input device 1012 ₁, inputdevice 1012 ₂, alphanumeric input device, keyboard, pointing device,mouse, etc.), one or more database interfaces (e.g., database interface1014 ₁, database interface 1014 ₂, etc.), one or more disk drive units(e.g., drive unit 1016 ₁, drive unit 1016 ₂, etc.), one or more signalgeneration devices (e.g., signal generation device 1018 ₁, signalgeneration device 1018 ₂, etc.), and one or more network interfacedevices (e.g., network interface device 1020 ₁, network interface device1020 ₂, etc.).

The disk drive units may include one or more instances of amachine-readable medium 1024 on which is stored one or more instances ofa data table 1019 to store electronic information records. Themachine-readable medium 1024 may further store a set of instructions1026 ₀ (e.g., software) embodying any one, or all, of the methodologiesdescribed above. A set of instructions 1026 ₁ may also be stored withinthe main memory (e.g., in main memory segment 1004 ₁). Further, a set ofinstructions 1026 ₂ may also be stored within the one or more processors(e.g., processor 1002 ₁). Such instructions and/or electronicinformation may further be transmitted or received via the networkinterface devices. Specifically, the network interface devices maycommunicate electronic information across a network using one or morecommunication links (e.g., communication link 1022 ₁, communication link1022 ₂, etc.). One or more network protocol packets (e.g., networkprotocol packet 1021 ₁, network protocol packet 1021 ₂, etc.) may beused to hold the electronic information (e.g., electronic data records)for transmission across the network.

The computer system 10A00 may be used to implement a client systemand/or a server system, and/or any portion of network infrastructure.Further, it is to be understood that various embodiments may be used asor to support software programs executed upon some form of processingcore (such as the CPU of a computer) or otherwise implemented orrealized upon or within a machine or computer readable medium. Amachine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine (e.g., acomputer). For example, a machine-readable medium includes read-onlymemory (ROM); random access memory (RAM); magnetic disk storage media;optical storage media; flash memory devices; or any other type ofnon-transitory media suitable for storing or transmitting information.

A module as used herein may be implemented using any mix of any portionsof the system memory, and any extent of hard-wired circuitry includinghard-wired circuitry embodied as one or more processors (e.g., processor1002 ₁, processor 1002 ₂, etc.).

FIG. 12B depicts a block diagram of a data processing system suitablefor implementing instances of the herein-disclosed embodiments. The dataprocessing system may include many more or fewer components than thoseshown.

The components of the data processing system may communicate electronicinformation (e.g., electronic data records) across various instancesand/or types of an electronic communications network (e.g., network1048) using one or more optical links, Ethernet links, wireline links,wireless links, and/or other electronic communication links (e.g.,communication link 1022 ₃, communication link 1022 ₄, etc.). Suchcommunication links may further use supporting hardware, such as modems,bridges, routers, switches, wireless antennas and towers, and/or othersupporting hardware. In some embodiments, the network 1048 may include,without limitation, the web (i.e., the Internet), one or more local areanetworks (LANs), one or more wide area networks (WANs), one or morewireless networks, and/or one or more cellular networks. The variouscommunication links transmit signals comprising data and commands (e.g.,electronic data records) exchanged by the components of the dataprocessing system, as well as any supporting hardware devices used totransmit the signals. In some embodiments, such signals are transmittedand received by the components at one or more network interface ports(e.g., network interface port 1023 ₁, network interface port 1023 ₂,etc.). In one or more embodiments, one or more network protocol packets(e.g., network protocol packet 1021 ₃, network protocol packet 1021 ₄,etc.) may be used to hold the electronic information comprising thesignals.

As shown, the data processing system may be used by one or moreadvertisers to target a set of users (e.g., user 1083 ₁, user 1083 ₂,user 1083 ₃, user 1083 ₄, user 1083 ₅, to user 1083 _(N)) comprising anaudience 1080 in various marketing campaigns. The data processing systemmay further be used to determine, by a computing platform 1030, variousattributes of such marketing campaigns. Other operations, transactions,and/or activities associated with the data processing system arepossible. Specifically, the users in audience 1080 may experience aplurality of online content 1053 transmitted through any of a pluralityof online channels 1076 (e.g., online display, search, mobile ads, etc.)to various computing devices (e.g., desktop device 1082 ₁, laptop device1082 ₂, mobile device 1082 ₃, and wearable device 1082 ₄). The users inaudience 1080 may further experience a plurality of offline content 1052presented through any of a plurality of offline channels 1078 (e.g., TV,radio, print, direct mail, etc.). The online content 1053 and/or theoffline content 1052 may be selected for delivery to the audience 1080based in part on certain instances of campaign specification datarecords 1074 (e.g., established by the advertisers and/or the computingplatform 1030). For example, the campaign specification data records1074 might comprise settings, rules, taxonomies, and other informationtransmitted electronically to one or more instances of online deliverycomputing systems 1046 and/or one or more instances of offline deliveryresources 1044. The online delivery computing systems 1046 and/or theoffline delivery resources 1044 may receive and store such electronicinformation in the form of instances of computer files 1084 ₂ andcomputer files 1084 ₃, respectively. In one or more embodiments, theonline delivery computing systems 1046 may comprise computing resourcessuch as a publisher web server 1062, a publisher ad server 1064, amarketer ad server 1066, a content delivery server 1068, and othercomputing resources. For example, the stimulus data record 1070 ₁presented to the users of audience 1080 through the online channels 1076may be transmitted through the communications links of the dataprocessing system as instances of electronic data records using variousprotocols (e.g., HTTP, HTTPS, etc.) and structures (e.g., JSON), andrendered on the computing devices in various forms (e.g., digitalpicture, hyperlink, advertising tag, text message, email message, etc.).The stimulus data record 1070 ₂ presented to the users of audience 1080through the offline channels 1078 may be transmitted as sensory signalsin various forms (e.g., printed pictures and text, video, audio, etc.).

The computing platform 1030 may receive instances of response datarecord 1072 comprising certain characteristics and attributes of theresponse of the users in audience 1080 to the stimulus data record 1070₁ and the stimulus data record 1070 ₂. For example, the response datarecord 1072 may describe certain online actions taken by the users onthe computing devices, such as visiting a certain URL, clicking acertain link, loading a web page that fires a certain advertising tag,completing an online purchase, and other actions. The response datarecord 1072 may also include information pertaining to certain offlineactions taken by the users, such as a purchasing a product in a retailstore, using a printed coupon, dialing a toll-free number, and otheractions. The response data record 1072 may be transmitted to thecomputing platform 1030 across the communications links as instances ofelectronic data records using various protocols and structures. Theresponse data record 1072 may further comprise data (e.g., computingdevice identifiers, timestamps, IP addresses, etc.) related to theusers' actions.

The response data record 1072 and other data generated and used by thecomputing platform 1030 may be stored in one or more storage devices1050 (e.g., stimulus data store 1054, response data store 1055,measurement data store 1056, planning data store 1057, audience datastore 1058, etc.). The storage devices 1050 may comprise one or moredatabases and/or other types of non-volatile storage facilities to storedata in various formats and structures (e.g., data tables 1082, computerfiles 1084 ₁, etc.). The data stored in the storage devices 1050 may bemade accessible to the computing platform by a query engine 1036 and aresult processor 1037, which may use various means for accessing andpresenting the data, such as a primary key index 1083 and/or othermeans. In one or more embodiments, the computing platform 1030 maycomprise a measurement server 1032 and an apportionment server 1034.Operations performed by the measurement server 1032 and theapportionment server 1034 may vary widely by embodiment. As an example,the measurement server 1032 may be used to analyze the stimuli presentedto the users (e.g., stimulus data record 1070 ₁ and stimulus data record1070 ₂) and the associated instances of response data record 1072 todetermine various performance metrics associated with a marketingcampaign, which metrics may be stored in the measurement data store 1056and/or used to generate various instances of the campaign specificationdata records 1074. Further, for example, the apportionment server 1034may be used to generate marketing campaign plans and associatedmarketing spend apportionment, which information may be stored in theplanning data store 1057 and/or used to generate various instances ofthe campaign specification data records 1074. Certain portions of theresponse data record 1072 might further be used by a data managementserver 1038 in the computing platform 1030 to determine various userattributes (e.g., behaviors, intent, demographics, device usage, etc.),which attributes may be stored in the audience data store 1058 and/orused to generate various instances of the campaign specification datarecords 1074.

In the foregoing specification, the disclosure has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the disclosure. Forexample, the above-described process flows are described with referenceto a particular ordering of process actions. However, the ordering ofmany of the described process actions may be changed without affectingthe scope or operation of the disclosure. The specification and drawingsare, accordingly, to be regarded in an illustrative sense rather than ina restrictive sense.

What is claimed is:
 1. An apparatus comprising: a model generator togenerate a learning model based on a correlation of stimulus data andresponse data, the stimulus data including stimuli presented to users ona channel or a sub-channel of the channel, the response data indicativeof user responses to the stimuli presented to the users, the learningmodel to predict user responses based on stimuli presented to the usersin the channel or the sub-channel, the correlation indicative of stimulicontributing to user responses at a channel or a sub-channel level; anattribution engine to determine a media spend plan based on the learningmodel and a budget, the media spend plan including an allocation of thebudget to stimuli corresponding to the channel or the sub-channel; and auser interface to display the media spend plan to a user and update themedia spend plan based on predictions of the learning model when theuser adjusts the budget or allocations of the media spend plan in theuser interface.
 2. The apparatus of claim 1, wherein the model generatoris to adjust the learning model by providing subsets of stimulus data tothe learning model and comparing responses predicted by the learningmodel to actual responses included in the response data.
 3. Theapparatus of claim 2, wherein the model generator is to adjust thelearning model using machine learning techniques.
 4. The apparatus ofclaim 1, further including a small signal correlation engine tocalculate correlation coefficients based on electronic data records thatinclude stimulus data and response data, the correlation coefficientsindicative of an amount a sub-channel contributed to a response or setof responses.
 5. The apparatus of claim 4, wherein the model generatoruses the correlation coefficients to improve sub-channel predictions ofthe learning model.
 6. The apparatus of claim 5, wherein the attributionengine uses the sub-channel predictions to adjust the allocationsdisplayed by the user interface.
 7. The apparatus of claim 1, whereinthe attribution engine determines the allocation of the budget to thestimuli corresponding to the channel or the sub-channel by calculatingstimulus contribution values for the stimuli based on the learning modeland applying the stimulus contribution values to the budget.
 8. Anapparatus comprising: means for generating to generate a learning modelbased on a correlation of stimulus data and response data, the stimulusdata including stimuli presented to users on a channel or a sub-channelof the channel, the response data indicative of user responses to thestimuli presented to the users, the learning model to predict userresponses based on stimuli presented to the users in the channel or thesub-channel, the correlation indicative of stimuli contributing to userresponses at a channel or a sub-channel level; means for attributing todetermine a media spend plan based on the learning model and a budget,the media spend plan including an allocation of the budget to stimulicorresponding to the channel or the sub-channel; and means fordisplaying the media spend plan to a user and update the media spendplan based on predictions of the learning model when the user adjuststhe budget or allocations of the media spend plan in the means fordisplaying.
 9. The apparatus of claim 8, wherein the means forgenerating is to adjust the learning model by providing subsets ofstimulus data to the learning model and comparing responses predicted bythe learning model to actual responses included in the response data.10. The apparatus of claim 9, wherein the means for generating is toadjust the learning model using machine learning techniques.
 11. Theapparatus of claim 8, further including means for correlating tocalculate correlation coefficients based on electronic data records thatinclude stimulus data and response data, the correlation coefficientsindicative of an amount a sub-channel contributed to a response or setof responses.
 12. The apparatus of claim 11, wherein the means forgenerating uses the correlation coefficients to improve sub-channelpredictions of the learning model.
 13. The apparatus of claim 12,wherein the means for attributing uses the sub-channel predictions toadjust the allocations displayed by the means for displaying.
 14. Theapparatus of claim 8, wherein the means for attributing determines theallocation of the budget to the stimuli corresponding to the channel orthe sub-channel by calculating stimulus contribution values for thestimuli based on the learning model and applying the stimuluscontribution values to the budget.
 15. A tangible computer readablestorage medium comprising instructions that, when executed, cause amachine to at least: generate a learning model based on a correlation ofstimulus data and response data, the stimulus data including stimulipresented to users on a channel or a sub-channel of the channel, theresponse data indicative of user responses to the stimuli presented tothe users, the learning model to predict user responses based on stimulipresented to the users in the channel or the sub-channel, thecorrelation indicative of stimuli contributing to user responses at achannel or a sub-channel level; determine a media spend plan based onthe learning model and a budget, the media spend plan including anallocation of the budget to stimuli corresponding to the channel or thesub-channel; and display the media spend plan to a user and update themedia spend plan based on predictions of the learning model when theuser adjusts the budget or allocations of the media spend plan in a userinterface.
 16. The tangible computer readable storage medium of claim15, wherein the instructions, when executed, cause the machine to adjustthe model by providing subsets of stimulus data to the learning modeland comparing responses predicted by the learning model to actualresponses included in the response data.
 17. The tangible computerreadable storage medium of claim 16, wherein the instructions, whenexecuted, cause the machine to adjust the learning model using machinelearning techniques.
 18. The tangible computer readable storage mediumof claim 15, wherein the instructions, when executed, further cause themachine to calculate correlation coefficients based on electronic datarecords that includes stimulus data and response data, the correlationcoefficients indicative of an amount a sub-channel contributed to aresponse or set of responses.
 19. The tangible computer readable storagemedium of claim 18, wherein the instructions, when executed, cause themachine to improve the sub-channel predictions of the learning modelusing the correlation coefficients.
 20. The tangible computer readablestorage medium of claim 15, wherein the instructions, when executed,cause the machine to determine the allocation of the budget to thestimuli corresponding to the channel or the sub-channel by calculatingstimulus contribution values for the stimuli based on the learning modeland applying the stimulus contribution values to the budget.